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1  Project  Objective 

The  objective  of  this  project  is  to  study  cooperative  behaviors  in  the  framework  of  mobile  / 
manipulatory  agents,  observer  agents  and  a  human  operator  for  semi- autonomous  material 
handling  in  unstructured  environments. 


2  Approach 

A  direct  impact  of  using  distributed  agents  (which  are  cheaper,  faster  and  lighter)  is  to 
perform  tasks  more  reliably,  in  that  a  broken  agent  can  be  replaced  by  another,  or  the 
remaining  agents  can  be  reconfigured  to  carry  out  a  task.  Conventionally,  physical  interac¬ 
tions  /contacts  between  agents  are  regarded  as  accidents  and  are  avoided  if  at  all  possible.  In 
this  approach  to  multi-agent  cooperation,  physical  interactions  are  positively  exploited  for 
the  dual  purposes  of:  (1)  cooperative  manipulation;  and  (2)  communication.  The  informa¬ 
tion  communicated  through  physical  interactions  includes  the  relative  position/orientation, 
and  interactive  forces/moments.  Sensing  gives  each  agent  independence,  while  communi¬ 
cation  is  a  prerequisite  to  cooperation.  The  assumption  is  that  different  agents,  including 
the  human  agent,  have  different  degrees  of  knowledge  and  views  of  the  world  with  varied 
spatiotemporal  resolution  and  capabilities  of  interaction.  Agents  ask  for  and  take  advice 
(communicated  information),  using  it  for  control  and  completion  of  their  task,  unless  they 
decide  their  sensory  input  is  more  appropriate  for  the  given  subtask. 

This  multi-agent  robotic  research  project  is  subdivided  into  subtasks,  each  of  which  lead  to 
a  Ph.D.  dissertation. 

1.  Human  Management  of  a  Hierarchical  System  for  the  Control  of  Multiple  Mobile  Robots. 
Julie  Adams. 

In  order  to  take  advantage  of  autonomous  robotic  systems,  and  yet  ensure  successful  comple¬ 
tion  of  all  feasible  tasks,  we  propose  a  mediation  hierarchy  in  which  an  operator  can  interact 
at  all  system  levels.  Robotic  systems  are  not  robust  in  handling  un-modeled  events.  Reac¬ 
tive  behaviors  may  be  able  to  guide  the  robot  back  into  a  modeled  state  and  to  continue. 
Reasoning  systems  may  simply  fail.  Once  a  system  has  failed  it  is  difficult  to  re-start  the 
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task  from  the  failed  state.  Rather,  the  rule  base  is  revised,  programs  altered,  and  the  task 
re-tried  from  the  beginning. 

Human-machine  interfaces  have  been  developed  for  applications  in  the  areas  of  nuclear  power 
plants,  aviation,  and  telerobotics,  however,  these  systems  are  generally  not  considered  au¬ 
tonomous  with  the  operator  providing  a  “supervisory”  role.  Typically,  the  human  operator 
controls  the  entire  task  execution. 

One  aspect  of  the  system  we  are  developing  permits  the  human  operator,  when  necessary,  to 
interact  with  all  levels  of  a  system  to  assist  with  process  errors.  This  interaction  encompasses 
all  areas  of  a  semi-autonomous  system  from  the  processes  which  would  be  considered  fully- 
autonomous  to  those  considered  telerobotic. 

Our  system,  MASC  -  Multiple  Agent  Supervisory  Control  system,  permits  the  agents  to  work 
autonomously  until  the  human  supervisor  is  requested  to  take  control  or  detects  a  problem. 
Our  design  strategy  is  to  develop  a  general  system  which  is  applicable  to  various  robotic 
systems.  We  combine  the  advantages  of  autonomous  systems  with  the  human’s  ability  to 
control  a  system  through  a  human-machine  interface.  MASC  provides  the  supervisor  with 
tools  to  interact  with  all  the  robotic  system  processing  levels.  These  interactions  may  correct 
corrupted  data  or  process  decisions  which  would  typically  cause  an  autonomous  system  to 
enter  an  unstable  state.  We  desire  to  create  a  more  comprehensive  semi-autonomous  system 
based  on  this  interaction  which  will  successfully  complete  the  execution  of  task  assignments. 

We  have  defined  four  hierarchical  levels  of  supervisory  interaction  with  the  various  robotic 
system  levels.  MASC  permits  the  supervisor  to  specify  task  assignments,  teleoperate  agents, 
display  sensory  data,  override  process  conclusions  and  reconfigure  the  system  during  sundry 
sensory  and  agent  failures. 

2.  A  Framework  for  Modeling  and  Verifying  Visually  Guided  Agents:  Design,  Analysis  and 
Experiments. 

Jana  Kosecka. 

The  successful  functioning  of  robotic  agents  in  a  dynamically  changing  environment  requires 
rich  sensory  input  and  advanced  sensory-motor  control.  The  robotic  agents  are  typically 
equipped  with  multiple  sensors  and  multiple  actuators.  The  interactions  between  agents 
and  the  environment  can  be  characterized  by  both  discrete  and  continuous  models.  The  ac¬ 
complishment  of  various  tasks  is  mediated  by  complex  coordination  and  interaction  between 
individual  sensing  and  control  strategies.  It  is  crucial  for  the  reliable  and  predictable  opera¬ 
tion  of  robotic  systems  that  the  design  be  within  a  structured  methodology  which  supports 
analysis  and  modularity. 

In  this  thesis  we  introduce  a  framework  for  modeling  and  verifying  autonomous  mobile  agents 
which  provides  a  unified  approach  to  the  design  and  analysis  of  systems  comprising  of  both 
continuous  and  discrete  event  components.  Our  framework  proposes  to  model  elementary 
sensing  and  control  strategies  by  making  appropriate  finite  state  machine  abstractions  which 
capture  the  discrete  event  aspects  of  continuous  models. 

In  order  to  achieve  the  desired  modularity  and  flexibility  we  define  a  Task  Specification  Lan¬ 
guage  for  composing  elementary  strategies  to  form  more  complex  missions.  The  robustness 
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and  reliability  of  individual  control  strategies  is  guaranteed  by  their  design  and  analysis  at 
the  continuous  level.  The  verification  of  the  discrete  event  interactions  between  strategies 
is  formulated  as  a  Supervisory  Control  Theory  Problem,  where  we  synthesize  a  supervisor 
which  serves  as  a  run-time  scheduler  and  monitor  of  the  ta.sk. 

The  overall  framework  is  verified  in  a  series  of  experiments  where  teams  of  visually  guided 
mobile  agents  are  engaged  in  various  navigation  tasks.  Within  our  framework,  we  formulate 
the  visually  guided  navigation  problem  and  propose  a  relational  model  of  the  environment 
embedded  in  a  finite  state  machine  structure  which  can  then  be  used  for  automatic  generation 
of  the  task  specification. 

3.  Sensor  Processing  for  Mobile  Robot  Localization,  Exploration  and  Navigation. 

Robert  Mandelbaum. 

In  the  context  of  a  mobile  robotic  agent,  we  describe  a  unified  framework  for  the  compe¬ 
tencies  of  localization,  navigation,  exploration  and  map-building.  In  this  work,  we  focus 
on  localization.  We  describe  the  design,  implementation,  testing  and  evaluation  of  data- 
processing  algorithms  for  three  sensor  modalities:  ultrasound,  stereo  vision  and  patterned 
light.  We  develop  a  sensor  model  for  each  modality.  In  each  case,  distinctive  features  of 
objects  in  the  field  of  view  are  extracted.  In  each  case,  the  output  of  the  algorithm  is  of  a 
form  which  facilitates  integration  within  the  framework,  and  hence  the  localization  of  the 
agent  in  a  partially-known  environrhent. 

We  delineate  a  computationally  efficient  method  for  sensor-based  localization  of  a  mobile 
robot  based  on  planar  features  extracted  using  ultrasound.  The  method  runs  in  time  linear 
in  the  number  of  detected  features,  both  for  establishing  correspondences  between  extracted 
and  map  features,  and  for  pose  estimation.  We  outline  the  extension  of  the  localization 
algorithm  to  integrate  the  other  sensor  modalities. 

Finally,  we  describe  a  method  which,  given  an  input  of  pose  measxirements  by  the  sensor- 
based  localization  algorithm,  produces  the  minimax  risk  fixed-size  confidence  set  estimate 
for  the  pose  of  the  mobile  agent.  The  work  in  this  dissertation  constitutes  the  first  applica¬ 
tion  to  the  mobile  robotics  domain  of  optimal  fixed-size  confidence-interval  decision  theory. 
The  approach  is  evaluated  in  terms  of  theoretical  capture  probability  and  empirical  capture 
frequency  during  actual  experiments  with  the  mobile  agent.  The  method  is  compared  to  sev¬ 
eral  other  procedures  including  the  Kalman  Filter  (minimum  mean  squared  error  estimate) 
and  the  identity  estimator  which  uses  the  measurement  itself  as  the  estimate.  The  minimax 
approach  is  found  to  dominate  all  the  other  methods  in  performance. 

4.  Local  and  Repeatable  Coordination  Schemes  for  Redundant  Manipulators. 

Chau- Chang  Wang. 

This  thesis  addresses  the  coordination  of  multiple  degrees  of  freedom  in  redundant  robot 
manipulators.  A  redundant  manipulator  is  one  that  possesses  more  degrees  of  freedom  than 
axe  necessary  to  accomplish  the  desired  tasks.  In  such  a  system,  the  inverse  kinematics 
problem  or  the  problem  of  decomposing  the  desired  end-effector  movement  into  joint  move¬ 
ments  is  underspecified.  Therefore,  it  is  possible  to  develop  algorithms  that  optimize  one  or 
more  performance  indices  while  ensuring  the  desired  end-effector  movement.  Most  previous 
algorithms  are  based  on  local  control  laws  which  do  not  guarantee  a  repeatable  solution. 
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In  other  words,  if  the  end-effector  traces  a  closed  trajectory,  such  laws  will  not  guarantee  a 
closed  trajectory  in  joint  space. 

The  focus  in  this  thesis  is  on  a  family  of  control  laws  that  can  be  obtained  by  modeling 
a  redundant  manipulator  as  a  hyperelastic,  articulated  kinematic  chain.  This  approach  is 
used  to  obtain  local  control  laws  as  well  as  control  laws  that  guarantee  repeatability.  The 
repeatable  control  scheme  is  plagued  by  algorithmic  singularities.  A  method  to  determine 
these  singularities  is  developed.  These  singularities  are  shown  to  correspond  to  instabilities 
in  the  corresponding  hyperelastic  structure  when  the  structure  is  subject  to  force  loading 
or  displacement  loading.  An  important  theorem  which  relates  the  instabilities  in  force  and 
displacement  loading  is  proved.  Finally,  a  screw-theoretic  formulation  yielding  compact  and 
computationally  efficient  solutions  is  developed. 

The  primary  application  considered  in  the  thesis  is  a  mobile  manipulator  and  the  problem 
of  coordinating  locomotion  and  majaipulation.  The  efficacy  of  the  proposed  control  schemes 
is  demonstrated  by  computer  simulations  as  well  as  experiments. 

5.  Control  and  Coordination  of  Locomotion  and  Manipulation  of  Wheeled  Mobile 
Manipulators. 

Yoshio  Yamamoto. 

In  this  thesis,  we  investigate  modeling,  control,  and  coordination  of  mobile  manipulators. 
A  mobile  manipulator  in  this  study  consists  of  a  robotic  manipulator  and  a  mobile  plat¬ 
form,  with  the  manipulator  being  mounted  atop  the  mobile  platform.  A  mobile  manipulator 
combines  the  dextrous  manipulation  capability  offered  by  fixed-ba^e  manipulators  eind  the 
mobility  offered  by  mobile  platforms.  While  mobile  manipulators  offer  a  tremendous  po¬ 
tential  for  flexible  material  handling  and  other  tasks,  at  the  same  time  they  bring  about  a 
number  of  challenging  issues  rather  than  simply  increase  the  structural  complexity.  First, 
combining  a  manipulator  and  a  platform  creates  redundancy.  Second,  a  wheeled  mobile 
platform  is  subject  to  nonholonomic  constraints.  Third,  there  exists  dynamic  interaction  be¬ 
tween  the  manipulator  and  the  mobile  platform.  Fourth,  manipulators  and  mobile  platforms 
have  different  bandwidths.  Mobile  platforms  typically  have  slower  dynamic  response  than 
manipulators.  The  objective  of  the  thesis  is  to  develop  control  algorithms  that  effectively 
coordinate  manipulation  and  mobility  of  mobile  manipulators. 

We  begin  with  deriving  the  motion  equations  of  mobile  manipulators.  The  derivation  present 
here  makes  use  of  the  existing  motion  equations  of  manipulators  and  mobile  platforms,  and 
introduces  the  velocity  and  acceleration  dependent  terms  that  accoimt  for  the  dynamic  in¬ 
teraction  between  manipulators  and  mobile  platforms.  Since  nonholonomic  constraints  play 
a  critical  role  in  control  of  mobile  manipulators,  we  then  study  the  control  properties  of  non¬ 
holonomic  dynamic  systems,  including  feedback  linearization  and  internal  dynamics.  Based 
on  the  newly  proposed  concept  of  preferred  operating  region,  we  develop  a  set  of  coordination 
algorithms  for  mobile  manipulators.  While  the  manipulator  performs  manipulation  tasks, 
the  mobile  platform  is  controlled  to  always  bring  the  configuration  of  the  manipulator  into  a 
preferred  operating  region.  The  control  algorithms  for  two  types  of  tasks  —  dragging  motion 
and  following  motion  —  are  discussed  in  detail.  The  effects  of  the  dynamic  interaction  are 
also  investigated. 
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To  verify  the  efficacy  of  the  coordination  algorithms,  we  conduct  numerical  simulations  with 
representative  task  trajectories.  Additionally,  the  control  algorithms  for  the  dragging  motion 
and  following  motion  have  been  implemented  on  an  experimental  mobile  manipulator.  The 
results  from  the  simulation  and  experiment  are  presented  to  support  the  proposed  control 
algorithms. 

6.  Stability  of  Grasped  Objects:  Beyond  Force  Closure. 

William  Stamps  Howard,  IV. 

This  dissertation  addresses  the  stability  and  compliance  of  grasped  objects  through  the 
derivation  of  the  stiffness  matrix  associated  with  each  grasp.  A  grasped  object  is  defined 
to  be  in  equilibrium  if  the  sum  of  all  forces  acting  on  a  body  equals  zero  and  the  sum  of 
all  moments  acting  on  a  body  also  equals  zero.  An  equilibrium  grasp  may  be  stable  or 
unstable.  Force  closed  grasps  are  a  well-known  subset  of  equilibrium  grasps  and  have  the 
important  property  of  being  stable.  However,  not  all  stable  grasps  are  force  closed,  including 
many  common  and  easily  obtainable  grasps.  In  this  dissertation,  a  general  framework  is 
established  for  the  determination  of  grasp  stability.  Simple,  easily  applicable  criteria  are 
derived  to  determine  the  stability  of  any  grasp,  including  the  special  but  important  limiting 
case  of  rigid  bodies. 

In  order  to  analyze  the  stability  of  grasps  with  multiple  contacts,  the  compliances  at  each 
contact  is  modeled.  The  model  includes  the  effect  of  such  parameters  as  curvatures  of  the 
contacting  surfaces,  kinematics  of  the  grasping  fingers,  internal  forces,  and  compliance  of 
the  fingers  and  joints  on  the  performance  of  the  system.  The  methodology  also  allows  the 
grasping  fingers  to  be  modeled  as  either  compliant  or  rigidly  fixed  in  space.  Expressions  are 
then  developed  for  the  changes  in  contact  forces  as  a  function  of  the  rigid  body  motion  of  the 
grasped  object.  From  this,  the  stability  of  a  grasp  is  shown  to  depend  on  the  local  curvature 
of  the  contacting  bodies,  eis  well  as  the  magnitude  and  arrangement  of  the  contact  forces.  A 
large  number  of  examples  are  presented,  including  applications  in  automated  fixturing. 

Finally,  the  asymmetry  of  Cartesian  stiffness  matrices  in  a  conservative  system  is  examined. 
Methods  of  differential  geometry  and  properties  of  Lie  groups  are  used  to  show  that  any 
conservative  system  subjected  to  a  non-zero  external  load,  the  resulting  Cartesian  stiffness 
matrix  may  be  asymmetric  whenever  the  basis  vector  utilized  are  not  derived  from  general¬ 
ized  coordinates,  in  any  inertial  or  body-fixed  reference  frame.  However,  a  reference  frame 
is  found  which  always  generates  a  symmetric  stiffness  matrix. 


3  The  Multi- agent  Testbed 

The  GRASP  Laboratory  has  developed  a  hardware-software  testbed  for  studying  tradeoffs 
between  sensing  and  communication  in  multi-agent  robotic  systems.  Tradeoffs  between  task 
execution  time,  task  execution  reliability,  system  cost,  and  system  reconfigurability  are  also 
considered.  Reconfiguring  multi-agent  robotic  systems  by  distributing  the  sensing  subtasks 
among  several  agents  and  using  low-bandwidth  communications  channels  to  communicate 
task-specific  symbolic  information  provides  the  system  user  with  a  significantly  more  reliable 
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system.  The  testbed  includes: 

1.  Two  observer  robots  (Figure  1)  with  multiple  non-contact  sensing  modalities  for  obstacle 
avoidance  and  navigation. 


(a)  (b) 


Figure  1:  (a)  Observation  agent  A  “sensorbot”  with  four  sensor  modalities:  ultrasound, 
stereo  pair,  patterned  light  and  dead  reckoning,  and  (b)  Observation  agent  B  “visionbot” 
with  two  sensor  modalities:  inverse  perspective  projection  and  dead  reckoning. 

With  “sensorbot,”  a  framework  has  been  developed  to  control  confluence  and  integration  of 
heterogeneous  sensor  data,  thereby  developing  sensor  modalities  in  a  modular  fashion  while 
enabling  cooperation  on  a  system  level.  Ultrasound  is  used  to  extract  geometric  information 
and  check  correspondence  with  world  map.  Agent  localization  is  embedded  in  formalism  of 
local  confldence  estimation,  which  predicts  tradeoffs  between  amount  of  precision  in  localiza¬ 
tion  afforded  by  detecting  more  features  versus  uncertainty  of  location  incurred  by  moving 
to  detect  the  features.  Reconfiguring  multi-agent  systems  by  distributing  sensing  subtasks 
and  using  low-bandwidth  channels  to  communicate  task-specific  information  yields  a  more 
reliable  system. 

With  “visionbot,”  a  framework  has  been  developed  in  which  the  Discrete  Event  System 
(DES)  formalism  is  connected  with  the  task  specification  language,  enabling  easy  incorpo¬ 
ration  of  heterogeneous  agents  and  the  human  operator.  Low-level  control  uses  continuous 
control  methods;  mid-level  control  uses  the  DES  formalism;  high-level  control  uses  the  task 
specification  language.  The  DES  supervisory  control  system  arbitrates  between  conflicting 
control  subtasks  for  a  given  task  requirement.  Our  approach  to  modeling  and  composition 
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of  agents’  behavior  and  communication  guarantees  overall  system  controllability,  provided 
assumptions  about  the  environment  are  correct.  If  not,  a  failure  mode  is  initiated,  i.e.,  error 
recovery  behaviors  are  invoked  or  the  human  operator  assists. 

2.  Two  mobile  manipulators  (Figure  2),  which  provide  for  cooperative  manipulation  and 
transport  of  non-compact  objects.  The  two  manipulatory  agents  employ  a  fixtureless  grasp 
to  carry  an  object.  By  using  multiple  manipulators  equipped  with  sensors,  agents  can 
greisp  arbitrarily  shaped  objects  without  specially-shaped  end  effectors.  We  have  developed 
algorithms  for  optimal  control  while  ceirrying  an  object  and  automatic  agent /manipulator 
reconfiguration . 


Figure  2;  Two  manipulatory  agents. 

3.  One  human  operator  interface  (Figure  3).  The  Multi-agent  Supervisory  Control  System 
provides  world  model  generation  and  maintenance,  and  graphical  display  of  multisensory 
data  from  robotic  agents,  enabling  the  human  operator  to  interact  with  all  levels  of  the 
robotic  system. 

The  testbed,  depicted  schematically  in  Figure  4,  is  flexible  enough  for  adding  or  turning  off 
processes,  and  studying  the  tradeoffs  between  sensing  versus  a  priori  models  and  their  effects 
on  performance,  including  statistical  analysis  of  process  outcomes  and  task  reliability.  The 
complexity  of  this  testbed  and  corresponding  benchmarking  of  team  cooperative  behaviors 
can  be  characterized  by  the  following  measures: 

•  We  have  4  mobile  platforms,  with  8  actuators,  2  manipulators,  6  different  sensory 
modalities,  and  4  processes  per  agent. 

•  The  human  operator  interface  and  cameraServer  provides  the  human  agent  with  images 
when  not  running  with  any  of  the  other  sensory  processes. 

•  The  high-level  software  of  the  human  operator  interface  alone  is  composed  of  34  *.c, 
and  33  *.h  files. 
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Figure  3:  The  Multi-agent  Supervisory  Control  System. 

•  The  human  operator  interface  currently  communicates  with  12  external  processes. 

•  We  run  three  different  hardware  platforms  (PC,  SGI  and  SUN). 

•  We  have  heterogeneous  connections  that  vary  for  different  tasks,  with  speeds  ranging 
from  5  HZ  up  to  250  HZ.  The  interconnection  possibilities  are  four  factorial,  selected 
automatically  based  on  the  dynamics  of  the  task. 


4  Project  Development 

4.1  June  1992  —  June  1993 

1.  Development  of  human  operator  interface  using  SGI’s  inventor  software.  This  approach 
did  not  work. 

2.  Development  of  hardware  and  basic  interfacing  software  for  ultrasound  sensor,  pat¬ 
terned  light  device  and  platform  controllers. 

3.  Design  of  novel  sensor  processing  and  clustering  strategies  with  real-time  performance 
for  ultrasound,  patterned  light  and  vision  modalities.  Strategies  address  the  tasks  of 
map-building,  basic  navigation  and  obstacle  avoidance. 

4.  Development  and  implementation  of  a  control  and  planning  algorithm  for  coordinating 
locomotion  and  manipulation  of  a  single  mobile  manipulator. 
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Figure  4:  Schematic  depiction  of  the  multi-agent  testbed. 

4.2  June  1993  —  June  1994 

1.  Development  of  the  human  operator  interface,  including:  (i)  initial  interface  devel¬ 
opment  using  Sayer’s  PENGUIN  software;  (ii)  display  of  raw  sonar  and  ultrasound; 
(iii)  development  of  teleoperation  to  create  commands;  (iv)  creation  of  overhead  pro¬ 
cess  to  add  objects  to  the  model;  (v)  integration  of  local  path  planner;  (vi)  formal 
development  of  the  mediation  hierarchy  theory;  (vii)  integration  of  Stanford’s  Global 
path  planner;  (viii)  integration  of  Obstacle  Avoidance  processes;  (ix)  development  of 
control  for  the  four  agents  (pausing,  stopping  or  continuing  command  execution);  and 
(x)  development  of  system  replay  for  problem  situations. 

2.  Development  of  algorithms  for  coordinating  two  manipulators  while  adapting  to  exter¬ 
nal  loading  and  maintaining  desired  contact  conditions  (e.g.,  force  closure). 

3.  Development  of  an  algorithm  for  controlling  a  mobile  manipulator  while  applying  a 
controlled  force  to  a  moving  object. 

4.  Development  of  a  procedure  for  allocating  end-effector  motion  between  the  base  and 
the  manipulator  to  solve  the  problem  of  kinematic  redundancy  for  mobile  manipulators. 

5.  Development  of  primitives  for  mobile  manipulator  controllers,  (e.g.,  Point-toJPoint 
shortest  path  generation). 
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6.  Further  design  and  implementation  of  elementary  sensor  strategies  with  real-time  per¬ 
formance  for  multiple  sensors  for  exploration,  building  world  models,  and  localization, 
and  preliminary  evaluation  of  performance. 

7.  Design  framework  for  integration  of  multiple  sensor  modalities  for  localization  and 
navigation. 

8.  Design,  development  and  implementation  of  a  large-scale  distributed  system.  Team 
consists  of  four  heavy-duty  agents  with  heterogeneous  capabilities  interacting  with 
the  human  agent.  The  team  boasts  seven  sensing  modalities,  including  two  vision- 
based  sensors,  a  patterned-light  device,  infra-red  proximity  sensors,  ultrasound  range 
detectors,  force/torque  sensors  and  odometry.  Each  team  member  consists  of  multiple 
processes  running  in  parallel. 

9.  Development  of  asynchronous  protocols  for  inter-process  as  well  as  inter-agent  com¬ 
munication  using  UNIX  sockets,  serial  lines  and  parallel  line  to  cope  with  the  vagaries 
of  a  generalized,  variegated,  heterogeneous  system. 

4.3  June  1994  -  April  1995 

1.  Development  of  the  human  operator  interface,  including:  (i)  further  development  of 
replay  mode;  (ii)  development  of  waypoint  specification  for  path  following;  (iii)  in¬ 
tegration  of  ultrasound  process;  (iv)  integration  of  puma  and  zebra  robots;  (v)  task 
execution  with  all  agents;  and  (vi)  attempt  to  integrate  stereo  process. 

2.  As  means  of  specifying  tasks  and  consequently  verifying  properties  of  the  system,  a 
process  oriented  task  specification  language  was  proposed  and  defined,  which  captures 
well  all  temporal  and  logical  dependencies  between  the  processes  on  the  distributed 
system,  and  enables  communication/cooperation  between  agents. 

3.  Continuing  testing  and  evaluation  of  performance  of  sensor  processing  algorithms. 

4.  Implementation  of  real-time  localization  algorithm  using  ultrasound  modality.  Design 
and  development  of  a  system  for  establishing  ground  truth  to  enable  quantitative 
evaluation  of  the  localization  algorithm.  Implementation  of  framework  to  combine 
data  from  multiple  modalities  for  the  task  of  localization. 

5.  Continuing  refinement  and  evolution  of  distributed  system  and  associated  communi¬ 
cation  protocols  to  accommodate  increasing  complexity. 

6.  Development  of  algorithms  that  enable  two  or  more  agents  to  march  in  formation 
while  maintaining  a  fixed  separation  distance  and  relative  orientation  and  a  method 
to  analyze  the  performance  of  such  multiagent  convoys. 

7.  Development  of  hybrid  agents  able  to  use  their  arms  for  locomotion  and  to  surmount 
obstacles. 
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4.4  April  1995  -  August  1996 

1.  Development  of  algorithms  that  enable  two  agents  to  reconfigure  while  cooperatively 
transporting  an  object. 

2.  Human  factors  testing  and  analysis  of  the  human  operator  interface. 

3.  Integration  and  experimental  verification  of  the  multi-agent  system,  involving  the  fol¬ 
lowing  experiments: 

(a)  Human  agent  prescribes  path  to  all  agents  and  they  follow  it.  No  obstacles  are 
in  the  path. 

(b)  Human  agent  prescribes  path  to  all  agents,  but  (unknown  to  the  human  agent) 
obstacle  appears  in  the  path. 

(c)  Human  agent  prescribes  path  to  all  agents,  which  includes  narrow  passageway 
requiring  the  two  manipulatory  agents  to  reconfigure. 

Four  mobile  autonomous  agents  marching  in  various  formations  is  depicted  in  Figure  5. 
(The  human  agent  supervision  and  assistance  is  not  shown.)  Agents  are  able  to  avoid 
obstacles,  carry  large  objects  and  reconfigure  themselves  while  holding  the  object  being 
carried.  This  scenario  has  been  experimentally  carried  out  in  the  GRASP  Laboratory. 
Furthermore,  this  scenario  was  remotely  demonstrated  at  the  DARPA  SISTO  Software 
Technology  and  Intelligent  Systems  Symposium  in  August,  1995  where  the  human 
operator  (at  the  demo  site  in  Chantilly,  VA)  supervised  and  assisted  the  four  robotic 
agents  in  the  GRASP  Lab  using  the  Multi-agent  Supervisory  Control  System. 

4.  Benchmark  team  cooperative  behavior. 

5  Conclusion 

5.1  Results 

•  We  have  implemented  a  cooperative  small  team  multi-agent  axchitecture  and  testbed 
for  transporting  non-compact  objects  through  obstacle-laden  environments. 

•  System  capabilities  include  interaction  between  a  human  agent,  two  mobile  manipula¬ 
tory  agents  and  two  observer  agents  with  multiple  non-contact  sensing  modalities  for 
path  planning  and  replanning,  obstacle  detection  and  avoidance,  payload  and  agent 
reconfiguration,  and  error  diagnosis  and  recovery. 

•  Four  mobile  autonomous  agents  are  able  to  march  in  various  formations  under  human 
supervision  and  assistance,  autonomously  avoiding  obstacles,  carrying  large  objects, 
and  reconfiguring  while  holding  the  object. 
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Figure  5:  Graphical  representation  of  three  experimental  scenarios,  (a)  Agents’  initial  loca¬ 
tions  and  orientations  are  known;  no  obstacles  encountered;  manipulatory  agents  undergo 
scheduled  reconfiguration  to  pass  through  doorway,  (b)  Agents’  relative  initial  locations 
and  orientations  are  known;  Agent  A  performs  global  localization;  an  obstacle  causes  devi¬ 
ation  from  planned  path;  manipulatory  agents  undergo  a  scheduled  reconfiguration  to  pass 
through  a  doorway,  (c)  Agents’  relative  initial  locations  and  orientations  are  known;  Agent 
A  performs  global  localization;  multiple  obstacles  warrant  unscheduled  reconfiguration  of 
manipulatory  agents. 
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5.2  Lessons  Learned 

•  Cooperative  testbeds  are  necessary  to  learn  about  and  test  the  robustness  of  system 
components. 

•  Design  and  development  of  a  mediation  hierarchy  as  part  of  the  human  agent  interface 
allows  the  human  agent  to  interact  with  the  whole  system  at  any  level  as  needed. 

•  The  human  agent  is  necessary  for  successful  operation  of  a  multi-agent  system  for 
cooperative  material  handling.  This  approach  exploits  human  expertise  for  aspects  of 
task  planning  and  monitoring,  and  error  recovery. 

•  Partitioning  robotic  agents  into  two  classes  provides  useful  degrees  of  freedom  in  ex¬ 
perimental  design,  allowing  for  systems  with  greater  ability  to  observe  agents  working 
near  obstacles  or  one  another. 

•  The  application  of  potential  functions  has  shown  the  value  of  this  approach  in  con¬ 
trolling  motion  of  multiple  vehicles  in  obstacle-laden  environments.  Motions  obtained 
exhibit  good  stability  and  smoothness. 

•  Embedding  the  overall  system  in  the  DES  formalism  connected  with  a  task  specification 
language  allows  for  easy  incorporation  of  heterogeneous  agents  and  a  human  operator. 

•  A  salient  aspect  of  our  DES  supervisory  control  system  is  its  ability  to  arbitrate  between 
conflicting  control  subtcisks  within  a  given  task  requirement. 

•  No  one  sensing  modality  can  provide  sufficient  data  about  the  environment  for  execu¬ 
tion  of  all  tasks;  thus,  sensory  integration  of  multiple  modalities  is  essential. 

•  One  observer  agent  can  localize  itself  and  other  agents  using  a  feature-based  strategy 
embedded  in  the  formalism  of  local  confidence  estimation,  which  predicts  tradeoffs 
between  precision  in  localization  afforded  by  detecting  more  features  versus  uncertainty 
of  location  incurred  by  moving  to  detect  features. 

•  Integrating  heterogeneous  computers  running  at  various  speeds  with  different  operating 
systems  and  inter- connections  is  very  difficult  and  should  never  be  underestimated! 

•  There  is  a  stiff  penalty  for  system  integration.  The  lack  of  standardized  off-the-shelf 
customized  subsystems  is  a  hindrance. 

•  Perception  is  the  main  bottleneck!  Assumptions  and  limitations  resulting  from  the 
system’s  perceptual  capabilities  determine  total  system  performance  and  influence  rea¬ 
soning  and  communication  capabilities. 
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